Eye gaze tracking calibration

ABSTRACT

A system and method for error estimation in an eye gaze tracking system in a vehicle may include an operator monitoring system providing measured eye gaze information corresponding to an object outside the vehicle and an external object monitoring system providing theoretical eye gaze information and an error in the measured eye gaze information based upon the measured eye gaze information and the theoretical eye gaze information.

INTRODUCTION

This disclosure is related to an operator monitoring system, for examplea motor vehicle driver monitoring system. More particularly, thisdisclosure relates to driver eye gaze determinations.

Eye gaze may generally refer to the direction that a driver's eyes arefixated at any given instant. Systems are known for operator eyetracking and for providing an operator eye gaze and may be used innumerous useful applications including detecting driver distraction,drowsiness, situational awareness and readiness to assume vehiclecontrol from an automated driving mode for example.

Eye tracking systems may exhibit errors in the eye gaze information theyprovide. One type of error may be generally categorized as a shift orbias in the indicated gaze. It is generally desirable to reduce oreliminate errors in gaze information from eye tracking systems.

SUMMARY

In one exemplary embodiment, an apparatus for error estimation in an eyegaze tracking system in a vehicle may include an operator monitoringsystem providing measured eye gaze information corresponding to anobject outside the vehicle, and an external object monitoring systemproviding theoretical eye gaze information and an error in the measuredeye gaze information based upon the measured eye gaze information andthe theoretical eye gaze information.

In addition to one or more of the features described herein, thetheoretical eye gaze information may be determined by the externalobject monitoring system based upon location information correspondingto the object outside the vehicle.

In addition to one or more of the features described herein, themeasured eye gaze information may include at least one of horizontalviewing angle information and vertical viewing angle information.

In addition to one or more of the features described herein, thelocation information may include depth of object information.

In addition to one or more of the features described herein, thelocation information may include object direction information.

In addition to one or more of the features described herein, thelocation information may include at least one of horizontal object angleinformation and vertical object angle information.

In addition to one or more of the features described herein, theexternal object monitoring system may include at least one forwardlooking camera.

In addition to one or more of the features described herein, theexternal object monitoring system may include at least one forwardlooking camera, and the location information may be determined based oncamera image data and relative displacement of the object outside thevehicle.

In another exemplary embodiment, a method for estimating an error in aneye gaze tracking system in a vehicle may include capturing, with aforward looking camera, a first exterior image at an earlier first timeand a second exterior image at a later second time. An object common toboth the first and second exterior images whose image position haschanged between the first exterior image and the second exterior imagemay be detected within each of the first and second exterior images.Respective first and second directions of the object may be determinedfor each of the first and second exterior images. A relativedisplacement of the object between the first time and the second timemay be determined. A first depth of the object at the first time and asecond depth of the object at the second time based on the firstdirection of the object, the second direction of the object, and therelative displacement of the object may be determined. A theoretical eyegaze of an operator of the vehicle looking at the object at a selectedone of the first time and the second time based on the corresponding oneof the first depth of the object and the second depth of the object andthe corresponding one of the first direction of the object and thesecond direction of the object may be determined. A measured eye gaze ofthe operator of the vehicle looking at the object at the selected one ofthe first time and the second time may be received from the eye gazetracking system. An error in the measured eye gaze of the operator basedupon the measured eye gaze and the theoretical eye gaze may bedetermined.

In addition to one or more of the features described herein, determiningthe theoretical eye gaze of the operator of the vehicle looking at theobject at one of the first time and the second time may be based on aseparation between the operator's eyes and the forward looking camera.

In addition to one or more of the features described herein, the objectmay be static and determining the relative displacement of the objectbetween the first time and the second time may be based upondisplacement of the vehicle between the first time and the second time.

In addition to one or more of the features described herein, the objectmay be dynamic and determining the relative displacement of the objectbetween the first time and the second time may be based upondisplacement of the vehicle between the first time and the second timeand displacement of the object between the first time and the secondtime.

In addition to one or more of the features described herein, determiningthe first depth of the object at the first time and the second depth ofthe object at the second time based on the first direction of theobject, the second direction of the object, and the relativedisplacement of the object may include representing the relativedisplacement of the object as a vector and solving an injective functioncomprising the vector, the first direction of the object, and the seconddirection of the object.

In addition to one or more of the features described herein, themeasured eye gaze of the operator may include at least one of ahorizontal viewing angle and a vertical viewing angle.

In addition to one or more of the features described herein, the firstdirection of the object and the second direction of the object may eachinclude at least one of a respective horizontal object angle and avertical object angle.

In addition to one or more of the features described herein, the methodmay further include recalibrating the eye gaze tracking system basedupon the determined error in the measured eye gaze of the operator.

In addition to one or more of the features described herein, determiningthe error may include at least one of a comparison of the measured eyegaze with the theoretical eye gaze, a statistical model, and a machinelearning model.

In addition to one or more of the features described herein, therecalibration may occur each vehicle cycle.

In yet another exemplary embodiment, an apparatus for error estimationin an eye gaze tracking system in a vehicle may include an operatormonitoring system providing measured eye gaze information correspondingto an object outside the vehicle, and an external object monitoringsystem determining location information corresponding to the object,determining theoretical eye gaze information based upon the determinedlocation information, and determining error in the measured eye gazebased upon the determined theoretical eye gaze information. Thedetermined error in the measured eye gaze may provide an errorestimation in the eye gaze tracking system.

In addition to one or more of the features described herein, thelocation information may include depth of object information and objectdirection information.

The above features and advantages, and other features and advantages ofthe disclosure are readily apparent from the following detaileddescription when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, advantages, and details appear, by way of example only,in the following detailed description, the detailed descriptionreferring to the drawings in which:

FIG. 1 illustrates an exemplary system, in accordance with the presentdisclosure;

FIG. 2 illustrates a forward looking view from within the passengercompartment of a host vehicle, in accordance with the presentdisclosure;

FIG. 3 illustrates an exemplary driving scene as may be captured in anexterior image by a forward looking camera as a host vehicle traverses aroadway, in accordance with the present disclosure;

FIG. 4A illustrates a simplified representation of overlaid images withrespect to a static object, in accordance with the present disclosure;

FIG. 4B illustrates a top down plan view representation of a drivingenvironment, in accordance with the present disclosure;

FIG. 5 illustrates determination of object depth information, inaccordance with the present disclosure;

FIG. 6A illustrates a simplified representation of an image with respectto a static object, in accordance with the present disclosure;

FIG. 6B illustrates a top down plan view representation of a drivingenvironment, in accordance with the present disclosure;

FIG. 7A illustrates a simplified representation of an image with respectto a static object, in accordance with the present disclosure;

FIG. 7B illustrates a side view representation of a driving environment,in accordance with the present disclosure; and

FIG. 8 illustrates an exemplary process flow for eye gaze errorestimation, in accordance with the present disclosure.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, its application or uses.Throughout the drawings, corresponding reference numerals indicate likeor corresponding parts and features. As used herein, control module,module, control, controller, control unit, electronic control unit,processor and similar terms mean any one or various combinations of oneor more of Application Specific Integrated Circuit(s) (ASIC), electroniccircuit(s), central processing unit(s) (preferably microprocessor(s))and associated memory and storage (read only memory (ROM), random accessmemory (RAM), electrically programmable read only memory (EPROM), harddrive, etc.) or microcontrollers executing one or more software orfirmware programs or routines, combinational logic circuit(s),input/output circuitry and devices (I/O) and appropriate signalconditioning and buffer circuitry, high speed clock, analog to digital(A/D) and digital to analog (D/A) circuitry and other components toprovide the described functionality. A control module may include avariety of communication interfaces including point-to-point or discretelines and wired or wireless interfaces to networks including wide andlocal area networks, on vehicle controller area networks and in-plantand service-related networks. Functions of control modules as set forthin this disclosure may be performed in a distributed controlarchitecture among several networked control modules. Software,firmware, programs, instructions, routines, code, algorithms and similarterms mean any controller executable instruction sets includingcalibrations, data structures, and look-up tables. A control module hasa set of control routines executed to provide described functions.Routines are executed, such as by a central processing unit, and areoperable to monitor inputs from sensing devices and other networkedcontrol modules and execute control and diagnostic routines to controloperation of actuators. Routines may be executed at regular intervalsduring ongoing engine and vehicle operation. Alternatively, routines maybe executed in response to occurrence of an event, software calls, or ondemand via user interface inputs or requests.

During roadway operation of a vehicle by a vehicle operator,semi-autonomously or fully-autonomously, the vehicle may be an observerin a driving scene which includes a driving environment including, forexample the roadway, surrounding infrastructure, objects, signs, hazardsand other vehicles sharing the roadway collectively referred to hereinas objects or targets. Objects may be static, such as road signage, ordynamic, such as another vehicle traversing the roadway. An observingvehicle may be referred to herein as a host vehicle. Other vehiclessharing the roadway may be referred to herein as target vehicles. Theterms driver and operator may be used interchangeably.

A host vehicle may be equipped with various sensors and communicationhardware and systems. An exemplary host vehicle 101 is shown in FIG. 1which illustrates an exemplary system 100, in accordance with thepresent disclosure. Host vehicle 101 may be a non-autonomous vehicle oran autonomous or semi-autonomous vehicle. The phrase autonomous orsemi-autonomous vehicle, as well as any derivative terms, broadly meansany vehicle capable of automatically performing a driving-related actionor function, without a driver request, and includes actions fallingwithin levels 1-5 of the Society of Automotive Engineers (SAE)International classification system. Host vehicle 101 may include acontrol system 102 including a plurality of networked electronic controlunits (ECUs) 117 which may be communicatively coupled via a busstructure 111 to perform control functions and information sharing,including executing control routines locally or in distributed fashion.Bus structure 111 may be a part of a Controller Area Network (CAN), orother similar network, as is well known to those having ordinary skillin the art. One exemplary ECU may include an engine control module (ECM)primarily performing functions related to internal combustion enginemonitoring, control and diagnostics based upon a plurality of inputsincluding CAN bus information. ECM inputs may be coupled directly to theECM or may be provided to or determined within the ECM from a variety ofwell-known sensors, calculations, derivations, synthesis, other ECUs andsensors over the bus structure 111 as well understood by those havingordinary skill in the art. Battery electric vehicles (BEV) may include apropulsion system control module (PSCM) primarily performing BEVpowertrain functions, including controlling wheel torque and electriccharging and charge balancing of batteries within a battery pack. Onehaving ordinary skill in the art recognizes that a plurality of otherECUs 117 may be part of the network of controllers onboard the hostvehicle 101 and may perform other functions related to various othervehicle systems (e.g. chassis, steering, braking, transmission,communications, infotainment, etc.). A variety of vehicle relatedinformation may be commonly available and accessible to all networkedECUs over the CAN bus, for example, vehicle dynamics information such asspeed, heading, steering angle, multi-axis accelerations, yaw, pitch,roll, etc. Another exemplary ECU may include an external objectcalculation module (EOCM) 113 primarily performing functions related tosensing the environment external to the vehicle 101 and, moreparticularly, related to roadway lane, pavement and object sensing. EOCM113 receives information from a variety of external object sensors 119and other sources. By way of example only and not of limitation, EOCM113 may receive information from one or more radar sensors, lidarsensors, ultrasonic sensors, two-dimensional (2D) cameras,three-dimensional (3D) cameras, global positioning systems,vehicle-to-vehicle communication systems, and vehicle-to-infrastructurecommunication systems, as well as from on or off board databases, forexample map and infrastructure databases. EOCM 113 may therefore haveaccess to position data, range data, rate data, and image data which maybe useful in the determination of roadway and target vehicleinformation, for example, roadway feature and target vehicle geometric,distance and velocity information, among others. Sensors 119 may bepositioned at various perimeter points around the vehicle includingfront, rear, corners, sides etc. as shown in the vehicle 101 by largedots at those positions. Sensor 119 positioning may be selected asappropriate for providing the desired sensor coverage for particularapplications. While sensors 119 are illustrated as coupled directly toEOCM 113, the inputs may be provided to EOCM 113 over the bus structure111 as well understood by those having ordinary skill in the art.Another exemplary ECU may include a driver monitoring module (DMM) 115tasked with monitoring the driver and/or passengers within the vehicleand primarily performing functions related to sensing the environmentwithin the vehicle 101 and, more particularly, related to driverinteraction with the vehicle, driver attentiveness, occupant posture andpositioning, seat-belt restraint positioning and other features andfunctions within the vehicle. The DMM 115 may receive information from avariety of sensors 121 and other sources. By way of example only and notof limitation, DMM 115 may receive information from one or moretwo-dimensional (2D) cameras and/or three-dimensional (3D) cameras,including infrared (IR) and/or near IR cameras.

Host vehicle 101 may be equipped with wireless communicationcapabilities shown generally at 123 which may be capable for one or moreof GPS satellite 107 communications, vehicle-to-vehicle (V2V) andvehicle-to-infrastructure (V2I) communications though known ad-hocnetworking, vehicle-to-pedestrian (V2P) communication, vehicle-to-cloud(V2C) communications such as through terrestrial radio (e.g. cellular)towers 105, or vehicle-to-everything (V2X) communications. In thepresent disclosure, reference to V2X is understood to mean any one ormore wireless communication capabilities connecting a vehicle toresources and systems off-board the vehicle including but not limited toV2V, V2I, V2P, V2C.

The description herein of the exemplary system 100 is not intended to beexhaustive. Nor is the description of the various exemplary systems tobe interpreted as being wholly required. Thus, one having ordinary skillin the art will understand that some, all and additional technologiesfrom the described exemplary system 100 may be used in variousimplementations in accordance with the present disclosure. One havingordinary skill in the art will appreciate that the described vehiclehardware is simply meant to illustrate some of the more relevanthardware components used with the present apparatus and methods and itis not meant to be an exact or exhaustive representation of vehiclehardware for implementing the present apparatus and methods.Furthermore, the structure or architecture of the vehicle hardware mayvary substantially from that illustrated in FIG. 1 (e.g., individualECUs may be integrated or otherwise combined with one another or withother equipment, as opposed to all being separate, stand-alonecomponents). Because of the countless number of potential arrangementsand for the sake of brevity and clarity, the vehicle hardware isdescribed in conjunction with the illustrated embodiment of FIG. 1, butit should be appreciated that the present system and method are notlimited to such embodiment.

FIG. 2 illustrates a forward looking view from within the passengercompartment 200 of vehicle 101. The passenger compartment 200 mayinclude one or more forward looking cameras 201 (hereafter camera 201)configured to capture a substantially forward looking driving scenethrough the windshield 203. Camera 201 may be mounted on dash panel 211,on or within rear view mirror 213, beneath the passenger compartmentheadliner, at an A-pillar or any other location providing anunobstructed view of the forward looking driving scene. Camera 201 mayalternatively be located external to the passenger compartment. Camera201 is mounted inside of the vehicle cabin and provides the presentsystem and method with camera image data. Although the followingexamples describe the camera 201 in the context of video cameras thatgenerate corresponding images or still frames, camera 201 may includeany suitable camera or vision system known or used in the industry, solong as it is capable of capturing exterior images, representationsand/or other information regarding the environment outside of thevehicle. Depending on the particular application, camera 201 mayinclude: a still camera, a video camera; a BW and/or a color camera; ananalog and/or digital camera; a wide and/or narrow field-of-view (FOV)camera; and may be part of a mono and/or stereo system, to cite a fewpossibilities. According to a non-limiting example, the vehicle hardwareincludes camera 201 that is a CMOS video camera and provides cameraimage data to EOCM 113 directly or via the bus structure 111. Camera 201may be a wide-angle camera (e.g., with a FOV of about 170° or more) sothat a full view or nearly full view of the relevant forward drivingscene may be obtained. The camera image data outputted by camera 201 mayinclude raw video or still image data (i.e., with no or littlepre-processing), or it may include pre-processed video or still imagedata in cases where the camera 201 has its own image processingresources and performs pre-processing on the captured images beforeoutputting them as camera image data.

FIG. 2 further illustrates the passenger compartment 200 including oneor more rear looking cameras 209 (hereafter camera 209) configured tocapture, at a minimum, images of a driver's eyes. Camera 209 may bemounted on dash panel 211, on or within rear view mirror 213, beneaththe passenger compartment headliner, at an A-pillar or any otherlocation providing an unobstructed view of the driver. A light source207 may be included as part of or separate from the camera 209. Lightsource 207 may preferably be an IR or near IR light source such as alight emitting diode (LED) as illustrated. According to a non-limitingexample, the camera 209 provides camera image data to the DMM 115directly or via the bus structure 111. The camera image data outputtedby camera 209 may include raw video or still image data (i.e., with noor little pre-processing), or it may include pre-processed video orstill image data in cases where the camera 209 has its own imageprocessing resources and performs pre-processing on the captured imagesbefore outputting them as camera image data. The DMM 115 determines thegaze direction of the driver from known techniques. For example, onetechnique is known as pupil center corneal reflection which may use thelight source 207 to illuminate the driver's eyes and the camera 209 tocapture images of the eyes including reflections of the light from oneor both eyes' corneas and pupils. A gaze direction may be determinedfrom the reflective geometries of the corneas and pupils and additionalgeometric features of the reflections. The DMM 115, including knownimage processing algorithms (including Artificial Intelligence (AI)),the camera 209 and light source 207 together make up an exemplary eyetracking system which may provide gaze information. Alternative eyetracking systems, including alternative eye tracking techniques andalgorithms, may be employed within the scope of the present disclosure.Gaze information may be represented by a vector or multiple componentvectors. For example, gaze information be represented by a vectorcorresponding to a horizontal angle component (i.e. left/right) and avector corresponding to an elevation angle component (i.e. up/down).

In accordance with the present disclosure, driver eye gaze may bemeasured and provided by an eye tracking system and is compared to atheoretical eye gaze determined based upon external object locationssensed by external sensors like cameras. Deviation of the driver'smeasured eye gaze from the theoretical eye gaze may be determined fromsuch comparisons and corrected as required. As used herein, the term“comparison” may mean any evaluation of measured eye gaze in view oftheoretical eye gaze and may include for example simple scalarcomparisons as well as statistical modeling or machine learning methods.

FIG. 3 illustrates an exemplary driving scene 300 as may be captured inan exterior image by forward looking camera 201 as the host vehicle 101traverses a roadway 315. Images from camera 201 may be boundary limitedat a top edge 303, bottom edge 305, left edge 307 and right edge 309.Horizontal image centerline 311 and vertical image centerline 313 areorthogonal image constructs and desirably substantially correspond tohorizontal directions and vertical directions with proper camera 201orientation and calibration. Similarly, top edge 303 and bottom edge 305desirably correspond to horizontal directions and left edge 307 andright edge 309 desirably correspond to vertical directions. Theexemplary driving scene of FIG. 3 includes a two-lane road 315 withadjacent lanes being distinguished by lane markers 317. Road shoulders319 are illustrated bounding the two-lane road 315. FIG. 3 representsthe overlay of two images separated in time, for example by severalseconds. Exemplary objects in the two images include a static object 321and a dynamic object 323 which, in the present embodiment, are a roadsign and preceding vehicle, respectively. The earlier captured image ofstatic object 321 at time t₀ is designated 321A and the earlier capturedimage of dynamic object 323 at time t₀ is designated 323A. The latercaptured images of static object 321 at time t₁ is designated 321B andthe later captured image of dynamic object 323 at time t₁ is designated323B. The earlier captured images of objects 321 and 323 at time t₀ areillustrated in FIG. 3 with broken lines, whereas the later capturedimages of objects 321 and 323 at time t₁ are illustrated in FIG. 3 withsolid lines.

FIG. 4A represents a simplified representation of the overlaid images ofFIG. 3 with respect to the static object 321. In this example, thestatic object 321 represents any static object or target that the camera201 and EOCM 113 may capture and process at different time stamps, forexample at earlier time t₀ and later time t₁ separated by severalseconds, preferably as the host vehicle 101 traverses a roadway. Timeseparation may be determined by such factors as vehicle speed and therelative displacement of objects between two time stamps. Processing byEOCM 113 may include, for example, image cropping, segmentation, objectrecognition, extraction and classification among other image processingfunctions as well known to those skilled in the art. Thus, earlier andlater captured images of static object 321 are labeled I_(A) and I_(B),respectively. Top edge 303, bottom edge 305, left edge 307, right edge309, and vertical image centerline 313 are also illustrated. Within theimages captured and overlaid as described, image distances W₁ and W₂ ofeach of the earlier and later captured images of the static object,I_(A) and I_(B), may be determined. Image distances W₁ and W₂ may bescaled in pixels, for example. Image distances W₁ and W₂ may be measuredfrom the vertical scene centerline 313. Vertical image centerline 313may be an arbitrary anchor point or reference for determination of imagedistances W₁ and W₂, it being understood that other references may beutilized including left and right edges 307 and 309. Similarly, a centerpoint may be an arbitrary point on the image of the objects fordetermination of image distances W₁ and W₂, it being understood thatother image reference points may be utilized including edges, corners,vehicle parts such as wheels, body panels, brake lights, etc. In thepresent disclosure, center points are used as reference points onobjects and images of objects and may be designated with labels A and Bin the drawings. An image plane 403 for object projection may bearbitrarily defined at some distance forward of the camera 201 such thatthe image distances W₁ and W₂ may be mapped to the image plane. Anexemplary image plane 403 is illustrated in FIG. 4B which is a top downplan view representation of the driving environment represented in FIG.3 and FIG. 4A. The image plane 403 is an arbitrary distance D, forexample 5 meters, from the forward looking camera 201. Vertical imagecenterline 313 orthogonally intersects image plane 403. Image distancesW₁ and W₂ may be mapped to the image plane in accordance with apredetermined scaling factor or function to return image plane distancesW₁′ and W₂′. Image plane distance W₁′ corresponds to the projected imageI_(A)′ and image plane distance W₂′ corresponds to the projected imageI_(B)′. From the image plane distances W₁′ and W₂′ and the distance D,the respective image plane angles α_(h)=tan⁻¹(D/W₁′) andβ_(h)=tan⁻¹(D/W₂′) may be determined. It is appreciated that the imageplane angles illustrated are in the horizontal plane and thus arelabeled with a subscripted h and may be referred to as horizontal imageplane angles or horizontal object angles. It is further appreciated thatthe horizontal image plane angles illustrated are the complementaryangles to the azimuth angles α_(h)′ and β_(h)′ corresponding to thestatic object 321 at the earlier and later time stamps relative to theforward looking camera 201 origin O and vertical image centerline 313.The horizontal image plane angles α_(h) and β_(h) or their complementaryazimuth angles α_(h)′ and β_(h)′ may define the direction of the staticobject 321 at earlier and later times t₀ and t₁, respectively. It isappreciated that the rays {right arrow over (OA)} and {right arrow over(OB)} represent the directions of the static object 321 at times t₀ andtime t₁, respectively.

With additional reference to FIG. 5, depth of objects may be determined.The relative displacement of the static object 321 is represented byvector {right arrow over (AB)} 503 which may be derived from the hostvehicle displacement from its first position at time t₀ to its secondposition at time t₁ from CAN bus vehicle position information andkinematic information. In the case of a static object, the vector {rightarrow over (AB)} 503 may simply be modeled as the inverse of the hostvehicle 101 displacement vector. In accordance with the presentdisclosure, information relating to position and motion of dynamicobjects may be based upon external object sensors 119. In oneembodiment, EOCM 113 may receive and derive kinematic information fromexternal object sensors 119, including one or more of range/rate capablesensors such as radar, lidar, ultrasonic and vision sensors, thatprovide data directly corresponding to static and dynamic objectposition and time derivatives thereof. That is, position, range,velocity, acceleration and jerk of an object within the reference frameof the host vehicle 101 may be available from such range/rate capablesensors. Additionally, it is known that range/rate sensors may alsoprovide moving object yaw rate, also within the reference frame of thehost vehicle 101. External object sensors 119 preferably provideposition, range, velocity, acceleration and jerk metrics in vehiclestandard longitudinally (X) and laterally (Y) resolved components.Otherwise, such resolving may be performed in EOCM 113. Depending uponthe degree of “at sensor” signal processing, downstream sensorprocessing may include various filtering. Additionally, where externalobject sensors 119 are numerically and/or topologically manifold,downstream sensor processing may include sensor fusing. Thus, it isappreciated that moving object kinematic information may include:longitudinal position (V_(t)P_(x)), velocity (V_(t)V_(x)) andacceleration (V_(t)A_(x)); lateral position (V_(t)P_(y)), velocity(V_(t)V_(y)) and acceleration (V_(t)A_(y)); and yaw rate (V_(t) {dotover (Ψ)}). Also within the reference frame of the host vehicle 101, CANbus data from host vehicle 101 may provide host vehicle kinematicinformation including: host vehicle longitudinal position (V_(h)P_(x)),velocity (V_(h)V_(x)) and acceleration (V_(h)A_(x)); host vehiclelateral position (V_(h)P_(y)), velocity (V_(h)V_(y)) and acceleration(V_(h)A_(y)); and host vehicle yaw rate (V_(h) {dot over (Ψ)}). Vehicleroll, pitch and vertical based information may also be included.

In another embodiment, host vehicle 101 and another vehicle on theroadway may be V2X capable allowing for the transmission of relevantinformation from the other vehicle for receipt by the host vehicle 101utilizing, for example, dedicated short-range communications (DSRC).Thus, the other vehicle may provide its kinematic information, includingV_(t)P_(x), V_(t)V_(x), V_(t)A_(x), V_(t)P_(y), V_(t)V_(y), V_(t)A_(y),and V_(t) {dot over (Ψ)}, vis-à-vis V2X communication of the othervehicle's CAN bus data. It is appreciated, therefore, that the othervehicle's kinematic information provided will be within the referenceframe of that other vehicle. One having ordinary skill in the art willrecognize that V2X information transfer may be directly between vehiclesor via one or more other neighboring nodes (surrounding vehicles orinfrastructure) through a V2V mesh network. Similarly, one havingordinary skill in the art will recognize that V2X information transfermay be by way of V2C routed communication, which may include additionalcloud resources and data enhancement and processing as well as extendthe communication distance between host vehicles 101 and other vehicles.

Relative displacements of dynamic objects, such as another vehicletraversing the roadway, therefore may be derived by additionallyaccounting for such other vehicle kinematics from time t₀ to time t₁,which information may be derived from host vehicle external objectsensors 119 or provided via V2X communications as described. A uniquevector fitting solution exists for vector {right arrow over (AB)} 503within the enclosed space between the rays {right arrow over (OA)} 411and {right arrow over (OB)} 413 representing the directions of thestatic object 321 at times t₀ and time t₁. Thus, an injective functionbetween the vector {right arrow over (AB)} 503 and the time stampeddirections (i.e. rays {right arrow over (OA)} 411 and {right arrow over(OB)} 413) exists and may be employed to determine the static object'spositions at respective times t₀ and t₁. From the objects' directionsand relative displacements, the static object's depth d_(A) from theorigin O at time t₀ and the static object's depth d_(B) from the originO at time t₁ are therefore known. Together, an object's depth anddirection information define the object's location.

FIG. 6A corresponds to FIG. 4A but only with respect to the earlierimage of static object 321 captured at time t₀. The exemplary imageplane 403 is illustrated in FIG. 6B as it was in FIG. 4B and is also atop down plan view representation of the driving environment. FIG. 6Bincludes for reference static object 321 depth d_(A), static object 321direction (i.e. ray {right arrow over (OA)} 411 and the correspondinghorizontal image plane angle or horizontal object angle α_(h)). Inaddition, FIG. 6B illustrates the driver's eyes 601. From the knowndepth d_(A) and direction, and further with known driver's eyes 601positional offset from the forward looking camera 201 origin O ifdesirable, a theoretical horizontal viewing angle γ_(h) may bedetermined. It is appreciated that, alternatively, the same proceduremay be followed utilizing the later image of static object 321 capturedat time t₁.

FIG. 7A corresponds to FIG. 6A with respect to the earlier image 321B ofstatic object 321 captured at time t₀. The exemplary image plane 403 isillustrated in FIG. 7B as it was in FIG. 6B; however, FIG. 7B is a sideview representation of the driving environment and is instructive of avertical analog to the horizontal representations illustrated in the topdown view of FIG. 6B. FIG. 7B includes for reference static object 321depth d_(A), static object 321 direction (i.e. ray {right arrow over(OA)} 411 and the corresponding vertical image plane angle or verticalobject angle α_(v)). In addition, FIG. 7B illustrates the driver's eyes601. From the known depth d_(A) and direction, and further with knowndriver's eyes 601 positional offset from the forward looking camera 201origin O if desirable, a theoretical vertical viewing angle γ_(v) may bedetermined. It is appreciated that, alternatively, the same proceduremay be followed utilizing the later image 321B of static object 321captured at time t₁.

It is to be appreciated that while a single pair of time separatedimages may sufficiently illustrate present subject matter, multiple suchpairs of time separated images may be employed. For example, multiplepairs of images clustered around a particular pair of time stamps maybeneficially be employed to effectively filter noise and disturbances(e.g. due to vehicle dynamics) from image signal information.Additionally, multiple pairs of images collected and evaluated across awide range of gaze angles as an object is tracked by an operator maybeneficially provide information as a function of gaze angle.

FIG. 8 illustrates an exemplary process flow for eye gaze errorestimation in accordance with the present disclosure. The process flowis suitable for use during active periods of eye gaze determinations bythe DMM 115 (i.e. measured eye gaze). Thus, error corrections may beadvantageously accomplished in real-time thus reducing the frequency ofoff-line calibration processes. Process 800 may be primarily implementedby EOCM 113 through execution of computer program code. However, certainsteps, for example calibration requests, may require actions on the partof the vehicle 101 operator which may be interpreted through varioususer interfacing including, for example, interfacing with a touch screendisplay in the cabin of vehicle 101, through a dialogue manager or otheruser interface. Additionally, the various computer implemented aspectsof process 800 may be executed within one or more other ECUs exclusivelyor in distributed fashion as previously disclosed and not necessarilylimited to exclusive execution by the EOCM 113. Process 800 isillustrated as a flow diagram with individual tasks in a substantiallylinear routine. One skilled in the art will understand that the processdescribed may be represented in alternative ways including state flowdiagrams and activity diagrams, for example. One skilled in the art alsounderstands that the various tasks in the process 800 flow diagram maybe implemented in different orders and/or simultaneously, andconsolidated or split.

Process 800 may be initiated (801) anytime the vehicle, including DMM115 and EOCM 113, is in an operationally ready state. One or more entryconditions may be evaluated (803) to determine whether process 800 isdesirable and capable. For example, a time or cycle threshold since aprior execution of process 800 may be a required entry condition, as maya manual driver calibration request for running the process 800. Process800 may run multiple times during a drive cycle or may be more limited,for example once during each drive cycle. Entry conditions may beperiodically evaluated (803) until satisfied, whereafter the process 800may proceed to read images (805) from the forward looking camera 201.Image reading (805) may include periodically capturing images at regularor variable time stamps or extracting still images from a continuousvideo feed buffer including time stamp information. Object detection(807) is next performed using the captured images. Image reading (805)and object detection (807) may include image cropping, segmentation,object recognition, extraction and classification among other imageprocessing functions. Object directions, for example horizontal andvertical image plane angles (horizontal and vertical object angles), aredetermined (809) for a pair of time separated images and object depthsdetermined (811) for the same pair of time separated images. Vehiclemovement and object movement during the time separation are used in thedetermination of relative object displacement (813). Object displacementinformation, object direction information and driver's eyes/cameraseparation information as needed are used to determine theoreticalhorizontal and vertical viewing angles for the driver's eyes for theobject at a given timestamp (815). The DMM 115 horizontal and verticalviewing angle information for the object at the same timestamp is thenprovided and compared with the theoretical horizontal and verticalviewing angles (817) from which an estimated bias or error andcorresponding corrections may be determined (819). Error determinationmay be carried out by for example by simple comparisons between the DMM115 provided viewing angle information and the theoretical viewing angleinformation. Alternatively, statistical models, such as regression, maybe employed using multiple incidents of DMM 115 provided information andcorresponding theoretical information. Multiple incidents may correspondto temporally clustered timestamps or more widely distributed timestamps(e.g. related to tracking an object through a wide range of viewingangles). The latter example may be particularly useful in developingdynamic error behaviors where errors may vary with viewing angle, forexample. Another error determination technique may include machinelearning which may be employed on vehicle, off vehicle via cloud or datacenter backroom processing, or a combination thereof. The correctionsmay then be provided to the DMM 115 for recalibration (821). The currentiteration of process 800 then ends (823).

Unless explicitly described as being “direct,” when a relationshipbetween first and second elements is described in the above disclosure,that relationship can be a direct relationship where no otherintervening elements are present between the first and second elements,but can also be an indirect relationship where one or more interveningelements are present (either spatially or functionally) between thefirst and second elements.

It should be understood that one or more steps within a method may beexecuted in different order (or concurrently) without altering theprinciples of the present disclosure. Further, although each of theembodiments is described above as having certain features, any one ormore of those features described with respect to any embodiment of thedisclosure can be implemented in and/or combined with features of any ofthe other embodiments, even if that combination is not explicitlydescribed. In other words, the described embodiments are not mutuallyexclusive, and permutations of one or more embodiments with one anotherremain within the scope of this disclosure.

While the above disclosure has been described with reference toexemplary embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substituted forelements thereof without departing from its scope. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the disclosure without departing from the essentialscope thereof. Therefore, it is intended that the present disclosure notbe limited to the particular embodiments disclosed, but will include allembodiments falling within the scope thereof.

What is claimed is:
 1. An apparatus for error estimation in an eye gazetracking system in a vehicle, comprising: an operator monitoring systemproviding measured eye gaze information corresponding to an objectoutside the vehicle; and an external object monitoring system providingtheoretical eye gaze information and an error in the measured eye gazeinformation based upon the measured eye gaze information and thetheoretical eye gaze information.
 2. The apparatus of claim 1, whereinthe theoretical eye gaze information is determined by the externalobject monitoring system based upon location information correspondingto the object outside the vehicle.
 3. The apparatus of claim 1, whereinthe measured eye gaze information comprises at least one of horizontalviewing angle information and vertical viewing angle information.
 4. Theapparatus of claim 2, wherein the location information comprises depthof object information.
 5. The apparatus of claim 2, wherein the locationinformation comprises object direction information.
 6. The apparatus ofclaim 2, wherein the location information comprises at least one ofhorizontal object angle information and vertical object angleinformation.
 7. The apparatus of claim 1, wherein the external objectmonitoring system comprises at least one forward looking camera.
 8. Theapparatus of claim 2: wherein the external object monitoring systemcomprises at least one forward looking camera; and wherein the locationinformation is determined based on camera image data and relativedisplacement of the object outside the vehicle.
 9. A method forestimating an error in an eye gaze tracking system in a vehicle,comprising: capturing, with a forward looking camera, a first exteriorimage at an earlier first time and a second exterior image at a latersecond time; detecting within each of the first and second exteriorimages an object common to both the first and second exterior imageswhose image position has changed between the first exterior image andthe second exterior image; determining for each of the first and secondexterior images respective first and second directions of the object;determining a relative displacement of the object between the first timeand the second time; determining a first depth of the object at thefirst time and a second depth of the object at the second time based onthe first direction of the object, the second direction of the object,and the relative displacement of the object; determining a theoreticaleye gaze of an operator of the vehicle looking at the object at aselected one of the first time and the second time based on thecorresponding one of the first depth of the object and the second depthof the object and the corresponding one of the first direction of theobject and the second direction of the object; receiving from the eyegaze tracking system a measured eye gaze of the operator of the vehiclelooking at the object at the selected one of the first time and thesecond time; and determining an error in the measured eye gaze of theoperator based upon the measured eye gaze and the theoretical eye gaze.10. The method of claim 9, wherein determining the theoretical eye gazeof the operator of the vehicle looking at the object at one of the firsttime and the second time is further based on a separation between theoperator's eyes and the forward looking camera.
 11. The method of claim9, wherein the object is static and determining the relativedisplacement of the object between the first time and the second time isbased upon displacement of the vehicle between the first time and thesecond time.
 12. The method of claim 9, wherein the object is dynamicand determining the relative displacement of the object between thefirst time and the second time is based upon displacement of the vehiclebetween the first time and the second time and displacement of theobject between the first time and the second time.
 13. The method ofclaim 9, wherein determining the first depth of the object at the firsttime and the second depth of the object at the second time based on thefirst direction of the object, the second direction of the object, andthe relative displacement of the object comprises representing therelative displacement of the object as a vector and solving an injectivefunction comprising the vector, the first direction of the object, andthe second direction of the object.
 14. The method of claim 9, whereinthe measured eye gaze of the operator comprises at least one of ahorizontal viewing angle and a vertical viewing angle.
 15. The method ofclaim 9, wherein the first direction of the object and the seconddirection of the object each comprise at least one of a respectivehorizontal object angle and a vertical object angle.
 16. The method ofclaim 9, further comprising recalibrating the eye gaze tracking systembased upon the determined error in the measured eye gaze of theoperator.
 17. The method of claim 9, wherein determining the errorcomprises at least one of a comparison of the measured eye gaze with thetheoretical eye gaze, a statistical model, and a machine learning model.18. The method of claim 16, wherein the recalibration occurs eachvehicle cycle.
 19. An apparatus for error estimation in an eye gazetracking system in a vehicle, comprising: an operator monitoring systemproviding measured eye gaze information corresponding to an objectoutside the vehicle; and an external object monitoring systemdetermining location information corresponding to the object,determining theoretical eye gaze information based upon the determinedlocation information, and determining error in the measured eye gazebased upon the determined theoretical eye gaze information; wherein thedetermined error in the measured eye gaze provides an error estimationin the eye gaze tracking system.
 20. The apparatus of claim 19, whereinthe location information comprises depth of object information andobject direction information.